Certified Test and Evaluation Professional (CTEP) Table of Specifications (aka CTEP Examination Blueprint)
I. REQUIREMENTS DEFINITION AND ANALYSIS _____________________ 20% Requirements definition and analysis for a “system, product, or capability” scheduled for testing is the critical baseline for test and evaluation professionals. If done correctly, it is a process which will increase the quality and reliability of the final system/product while minimizing costly rework resulting from requirements errors found engineering development and more importantly, during test and evaluation. Requirements must be formally documented, measurable, testable and traceable with both forward and backwards traceability. Given these basic criteria are met, the tester can plan and validate through testing whether or not the designed system or product meets and/or delivers the solution, services and actions that the customer expected (i.e. the requirements). This category may include questions on: A. Capabilities assessment B. Developing and mapping requirements to measures, metrics, and test objectives. C. Integrated system design D. International/national/local regulations as applicable E. Requirements decomposition process F. Safety standards G. Test methodology development (verification matrix) H. Test requirements generation and analysis process I. Writing good program/system T&E requirements II. DATA COLLECTION AND ANALYSIS _______________________________ 20% The proper selection, implementation, and operation of appropriate test data collection and analysis methods are of fundamental importance when determining the optimal experimental test design for a test program. A well-developed test data collection method coupled with appropriate data analysis approaches serves to produce an efficiently conducted and statistically robust set of test data results. A T&E Professional must: Be able to select appropriate Test Data Collection and Analysis Methodologies based upon the requirements of the system under test, and to ensure that the chosen methods can be executed within a given program’s scope of budget and schedule and within acceptable risk and resource constraints; Understand and effectively leverage the advantages, limitations, and current state of the practices of available data collection tools and data analysis methods; and, Understand the implications of emerging data collection and analysis methods and effectively implement such methods in test data analysis plans. This category may include questions on: A. Configuration management B. Data collection methodologies C. Independent data verification, validation, and accreditation D. Instrumentation and calibration E. Mathematics and statistics F. Measurement accuracy and precision G. Measurement process H. Operations research and other types of analysis I. Root cause analysis J. Statistics principles and tools (e.g. DOE) K. Test data management III. TEST TOOLS AND METHODOLOGIES _____________________________15% The selection of appropriate Test Tools and Methods is critical to conducting an effective and efficient test program. A T&E Professional must be able to select appropriate Test Tools and Methodologies based upon the tool's or methodology's ability to provide the requisite data within the needed environments, under acceptable risk and within resource constraints. They should understand the advantages, limitations, and "state of the practice" for a wide range of testing venues including, but not limited to: Simulation environments, laboratory, hardware/software-in-the-loop, prototype and full scale testing. The T&E Professional should understand the implications of emerging technologies and safety considerations on Test Tool and Methodology selection. This category may include questions on: A. Computer operating systems and software B. Design for testability C. Diverse requirements traceability tools D. Human and team dynamics/behaviors E. Instrumentations F. Modeling, simulation, stimulation, and prototyping G. Non-Destructive (NDI) testing techniques H. Relevant emerging technologies and methods I. Risk assessment and management J. Safety tools management K. Scientific methods L. Systems engineering principles and practices M. T&E best practices N. T&E methodologies, methods, and practices O. Teamwork and collaboration P. Test automation: Strategies and architectures Q. Test tool evaluation and selection IV. TEST TYPES ______________________________________________________ 15% The T&E Professional understands that the primary purpose of testing is to mitigate and categorize risks associated with the System Under Test (SUT) and understands the differences between the various risks that may be associated with the SUT (e.g. The risks associated with an engine failure are greatly different if that engine is in a ground vehicle or in an aircraft). While the same Test Type may be utilized in testing the engines, the instrumentation, length of test, data reduction, post-test examinations of the engine, etc. may differ greatly. The level of the requirement (e.g. Is this a desired or required capability? Is this a performance or Regulatory requirement?) may vary from SUT to SUT although the overall item descriptions may be similar. The T&E Professional understands the various test types, understands the various levels that may be associated with testing each parameter, and understands the impacts on the overall test. This category may include questions on: A. Agile testing B. Automated testing C. Commercial Off-the-shelf (COTS) testing D. Common human factors considerations E. Compliance testing F. Distributed testing G. End-to-End testing H. Environmental testing I. Interoperability testing J. Types of testing in various phases of development (e.g. component, integrated, developmental, operational, procurement, production) K. Software testing L. Subsystems testing M. System under test N. System-of-Systems testing O. Systems architectures V. PLANNING _______________________________________________________ 10% While the common belief may be that Test Planning is the process done just before Test Conduct, the T&E Professional understands that the best test planning starts early in the Program Development Planning phase. A T&E Professional with a good understanding of the various types and methodologies of testing over the entire range of testing (laboratory, chamber, component, system, system of systems, etc.) is a productive influence on the overall program design insuring that the Program Manager does not sign up for a “good idea” that cannot be sufficiently or adequately tested. With that early involvement of the T&E Professional in the Program Development Planning process, then during the actual planning of a test the T&E Professional has an understanding of the details of the products operation and therefore can effectively conduct the necessary Test Planning to accomplish the desired objectives. This category may include questions on: A. Common T&E contractual elements and terms B. Concurrence and approval hierarchies/stakeholders C. Development and T&E life cycle D. Key steps and major activities in T&E process E. Program/system T&E development process F. Project management G. Scheduling and project milestones H. T&E organizational structure I. T&E resources and capabilities J. T&E master/strategy plan K. Test planning and strategy development L. Test scenario development M. Typical relevant program/system documentation VI. TEST EXECUTION ________________________________________________ 10% Test Execution begins with a review to ensure the readiness of the test article to begin the test, and that the facilities, instrumentation, data recording devices, test procedures, and personnel are ready to conduct the test. The roles and responsibilities of members of the test team must be understood. During the conduct of the test, the test discipline of strictly following the test plan or procedure is of the utmost importance to ensure documented, repeatable tests. The T&E professional should be able to develop and implement contingency plans detailing pause-test or stop-test points when anomalies occur. This category may include questions on: A. Contingency planning (e.g. Stop/pause test decision point, alternative strategies) B. Design readiness review C. “Go/No Go” decision points D. Test discipline (e.g. Adherence to the test plan/requirements, version/configuration control) E. Test readiness confirmation F. Test team roles and responsibilities VII. REPORTING ______________________________________________________ 10% The purpose of Test and Evaluation is to provide decision-quality information. The T&E Professional must be able to turn "data" into useable information and present that information clearly, effectively, and ethically to the decision maker. This requires the T&E Professional to understand the relationship of the test and various types of reports (e.g. progress, incident, final, etc.) to the decision being made. The T&E Professional should be able to effectively employ various presentation media (e.g., written, oral, photos, multi-media) and techniques (e.g. graphical, tabular, pictorial, etc.) to communicate test results to the decision maker. The T&E Professional has an ethical responsibility to present results in an unbiased objective manner and to follow reporting guidelines and policies articulated by their respective organizations as well as offer a recommendation as to how well the item meets its performance criteria. This category may include questions on: A. Ethical Issues B. Presentation methods (e.g. written, verbal, multi-media) C. Presentation tools (e.g. briefings, documents, photos, videos, software applications) D. Types of T&E reports (e.g. plan approval, final, quick-look, incident, deficiency) Basic Skills Required of CTEPs BASIC SCIENCE SKILLS BASIC MATH SKILLS IT SKILLS ENGINEERING SKILLS PROJECT MGMT SKILLS Biology Addition Business software (e.g. word processing, spreadsheet, project management, presentation, graphics) Test engineering processes and procedures Accounting Benchmark allocation Chemistry Algebra Basic design principals of both digital and analog circuitry Budgeting Environmental Division Contracts and contracting Physics Geometry Deliverables Configuration management Estimating Multiplication Maintaining data integrity and security Data analysis Personnel/team management Probability Functional analysis Modeling and simulation Project documentation Statistics Measurement techniques and concepts Resourcing (materials, equipment, personnel) Subtraction Trigonometry Requirements validation and verification Scheduling Work break-down structure